Question Answering Challenge (QAC-1): An Evaluation of Question Answering Task at NTCIRWorkshop 3

In this paper we describe a question answering task, called Question Answering Challenge (QAC), and its first evaluation (QAC1). This was carried out as a task of NTCIR Workshop 3 in October 2002. In QAC, we aimed to encourage the development of practical QA systems in a general domain and focus on research of user interaction and information extraction. Developments of evaluation method of question answering system and information resources for evaluation were also purpose of QAC. We have defined three kinds of task in QAC, which require five possible answers (Task 1), only one answer (Task 2) and one answer to related question (Task 3). We have prepared 200 questions for Task 1 and Task 2 and 40 questions for Task 3 at Formal Run and about 900 questions for additional run. We have conducted Dry Run and Formal Run evaluation and finally there are sixteen participants (two of them are from task organizer) at QAC1.